Optimisation of on–line principal component analysis

Various techniques, used to optimise on-line principal component analysis, are investigated by methods of statistical mechanics. These include local and global optimisation of node-dependent learning-rates which are shown to be very efficient in speeding up the learning process. They are investigated further for gaining insight into the learning rates’ timedependence, which is then employed for devising simple practical methods to improve training performance. Simulations demonstrate the benefit gained from using the new methods. 1.Introduction The investigation of unsupervised on-line learning algorithms [1, 2] by means of statistical mechanics has been shown to be a useful tool for gaining insight on the training dynamics [3]. In contrast to batch algorithms whereby all available examples are considered simultaneously for calculating a single student parameters update, on-line updates are carried out after the presentation of each single data point (for an overview on current on-line methods in neural networks see [4]). This update is proportional to a learning rate η that has to be smaller than a critical value to make learning possible [5]. Successful learning is only possible if the learning rate is relatively small which, at the same time, means that many update steps are needed. Therefore, a relatively large rate is needed at the beginning and a smaller one later on; perfect learning is only possible if η → 0 at